System and method for processing an image

ABSTRACT

A system and method for processing an image including the steps of receiving an input image having a plurality of pixels, wherein each of the plurality of pixels have one or more pixel characteristics; and processing the input image to generate an enhanced image by applying a pixel/image relationship to each of the plurality of pixels of the input image, wherein the pixel/image relationship is arranged to adjust the one or more pixel characteristics of each of the plurality of pixels of the input image.

TECHNICAL FIELD

The present invention relates to a system and method for processing animage, and particularly, although not exclusively to a system and methodfor enhancing images or image signals.

BACKGROUND

The capturing, viewing and processing of photographs, videos and otherimages are common activities amongst photographers, media producers orsocial media users. With commonly accessible image or video capturingequipment such as digital cameras, action cameras or smart devices (e.g.smartphones) with cameras, images and videos have become a common andexpected form of media for communications and the sharing of ideas orknowledge between different people.

Despite advances in photography and the use of photographic equipment,environmental conditions may nonetheless limit the quality of the videosor images captured in real world conditions. Insufficient lighting is aproblem that users suffer, which in turn causes degradations invisibility, brightness, contrast and details. In turn, whenenvironmental conditions turn undesirable, such as in low lightconditions or in environments where there are strong variations inlighting conditions, a video or image that is captured by a user mayappear to be aesthetically poor or undesirable.

Similarly, due to the widespread usage of image capturing or imagebroadcasting equipment, there is a large variety of specificationsbetween the different equipment used by individual users. In turn, ahigh quality image may have been captured by a user with equipment of asuperior specification, but when it is broadcast or viewed by anotherend user with equipment that has an inferior or differentspecifications, the image or video may in turn be presented poorly or ofa lesser quality due to the limitations of the transmission or displayequipment.

SUMMARY OF THE INVENTION

In accordance with a first aspect of the present invention, there isprovided a method for processing an image comprising the steps of:

-   -   receiving an input image having a plurality of pixels, wherein        each of the plurality of pixels have one or more pixel        characteristics; and,    -   processing the input image to generate an enhanced image by        applying a pixel/image relationship to each of the plurality of        pixels of the input image, wherein the pixel/image relationship        is arranged to adjust the one or more pixel characteristics of        each of the plurality of pixels of the input image.

In an embodiment of the first aspect, the pixel/image relationship isassociated with a relationship between the one or more ofcharacteristics of each of the plurality of pixels of the input imageand a visual presentation of the input image.

In an embodiment of the first aspect, the one or more characteristics ofeach of the plurality of pixels are associated with the exposure levelsof each of the plurality of pixels.

In an embodiment of the first aspect, the pixel/image relationshipincludes a mapping of the one or more characteristics of each of theplurality of pixels relative to the one or more characteristics of eachof the other pixels of the input image.

In an embodiment of the first aspect, the exposure levels include tones,contrasts or color shifts of the pixel.

In an embodiment of the first aspect, by applying the pixel/imagerelationship to each of the plurality of the pixels of the input image,a dynamic range of the input image is adjusted.

In an embodiment of the first aspect, the pixel/image relationship isapplied iteratively to each of the plurality of pixels are processed togenerate the enhanced image.

In an embodiment of the first aspect, the pixel/image relationshipincludes at least one adjustable parameter arranged to adjust amagnitude of the application of the pixel/image relationship to each ofthe plurality of pixels.

In an embodiment of the first aspect, the magnitude of the applicationof the pixel/image relationship applies to the one or morecharacteristics of each of the plurality of pixels.

In an embodiment of the first aspect, the at least one adjustmentparameter is adjustable by a learning network.

In an embodiment of the first aspect, the learning network is aconvolution neural network (CNN).

In an embodiment of the first aspect, the learning network is trainedwith a reference data set.

In an embodiment of the first aspect, the learning network is trainedwith one or more image quality loss processes.

In an embodiment of the first aspect, the one or more image quality lossprocesses include spatial consistency loss, exposure control loss, colorconstancy loss, illumination smoothness loss or any combination thereof.

In an embodiment of the first aspect, the one or more image quality lossprocesses are used when the reference data set is not available.

In an embodiment of the first aspect, the image/pixel relationship isrepresented by F(I(x);α)=I(x)+αI(x)(1−I(x)) where

-   -   x denotes the pixel coordinates,    -   F(I(x);α) is the enhanced image,    -   I(x) , α∈[−1,1] is the adjustable parameter and wherein each        pixel is normalized to [0,1].

In an embodiment of the first aspect, the spatial consistency loss isrepresented by

$L_{spa} = {\frac{1}{K}{\sum\limits_{i = 1}^{K}\;{\sum\limits_{j \in {\Omega{(i)}}}\left( {{{Y_{i} - Y_{j}}} - {{I_{i} - I_{j}}}} \right)^{2}}}}$

where K is the number of cantered region i, Ω(i) is the four neighboringregions cantered at the region i. Y and I represent an average intensityvalue of a local region in the enhanced result and input image,respectively.

In an embodiment of the first aspect, the exposure loss control isrepresented by:

$L_{\exp} = {\frac{1}{M}{\sum\limits_{k = 1}^{M}\;{{Y_{k} - E}}}}$

where M represents the number of nonoverlapping local regions, Y is theaverage intensity value of a local region in the enhanced image.

In an embodiment of the first aspect, the color Constancy Loss isrepresented by

${L_{col} = {\sum\limits_{\forall{{({p,q})} \in {\mathcal{s}}}}\left( {J^{p} - J^{q}} \right)^{2}}},{ɛ = \left\{ {R,G,B} \right\}}$

where J^(p) denotes the average intensity value of p channel of theenhanced result, (p,q) represents a pair of color channel.

In an embodiment of the first aspect, the illumination smoothness lossis represented by

${L_{{tv}\;\alpha} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}\;{\sum\limits_{c \in {\mathcal{s}}}\left( {{\nabla_{x}\alpha_{n}^{c}} + {\nabla_{y}\alpha_{n}^{c}}} \right)^{2}}}}},{ɛ = \left\{ {R,G,B} \right\}}$

where N is the number of iterations, a is the curve parameter map, ∇_(x)and ∇_(y) represent the horizontal and vertical gradient operations.

In an embodiment of the first aspect, the input image is a SDR signaland the enhanced image is a HDR signal.

In accordance with a second aspect of the present invention, there isprovided a system for processing an image comprising:

-   -   an image gateway arranged to receive an input image having a        plurality of pixels, wherein each of the plurality of pixels        have one or more pixel characteristics; and,    -   an enhancement engine arranged to process the input image to        generate an enhanced image by applying a pixel/image        relationship to each of the plurality of pixels of the input        image, wherein the pixel/image relationship is arranged to        adjust the one or more pixel characteristics of each of the        plurality of pixels of the input image.

In an embodiment of the second aspect, the pixel/image relationship isassociated with a relationship between the one or more characteristicsof each of the plurality of pixels of the input image and a visualpresentation of the input image.

In an embodiment of the second aspect, the one or more characteristicsof each of the plurality of pixels are associated with the exposurelevels of each of the plurality of pixels.

In an embodiment of the second aspect, the pixel/image relationshipincludes a mapping of the one or more characteristics of each of theplurality of pixels relative to the one or more characteristics of eachof the other pixels of the input image.

In an embodiment of the second aspect, the exposure levels includetones, contrasts or color shifts of the pixel.

In an embodiment of the second aspect, by applying the pixel/imagerelationship to each of the plurality of the pixels of the input image,a dynamic range of the input image is adjusted.

In an embodiment of the second aspect, the pixel/image relationship isapplied iteratively to each of the plurality of pixels are processed togenerate the enhanced image.

In an embodiment of the second aspect, the pixel/image relationshipincludes at least one adjustable parameter arranged to adjust amagnitude of the application of the pixel/image relationship to each ofthe plurality of pixels.

In an embodiment of the second aspect, the magnitude of the applicationof the pixel/image relationship applies to the one or morecharacteristics of each of the plurality of pixels.

In an embodiment of the second aspect, the at least one adjustmentparameter is adjustable by a learning network.

In an embodiment of the second aspect, the learning network is aconvolution neural network (CNN).

In an embodiment of the second aspect, the learning network is trainedwith a reference data set.

In an embodiment of the second aspect, the learning network is trainedwith one or more image quality loss processes.

In an embodiment of the second aspect, the one or more image qualityloss processes include spatial consistency loss, exposure control loss,color constancy loss, illumination smoothness loss or any combinationthereof.

In an embodiment of the second aspect, the one or more image qualityloss processes are used when the reference data set is not available.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will now be described, by way ofexample, with reference to the accompanying drawings in which:

FIG. 1 is a schematic diagram of a computer system arranged to operateas a system for processing an image in accordance with one embodiment ofthe present invention;

FIG. 2 is a block diagram illustrating an embodiment of a system forprocessing an image;

FIG. 3 is a block diagram illustrating an embodiment of the imageenhancement engine of FIG. 2;

FIG. 3A shows diagrams (a) to (e) showing an example pixel/imagerelationship with different parameters and number of iterations;

FIG. 3B shows images (a) to (e) showing examples of the best-fittingcurve parameter maps of RGB channels;

FIG. 4 is a block diagram illustrating an embodiment of the imagecharacteristic parameters estimation network of FIG. 3;

FIG. 5 shows images (a) to (f) showing an example enhancement of aninput image without any one of image quality loss functions of FIG. 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Referring to FIG. 1, there is illustrated a computing device 100 whichis arranged to be implemented to provide a method for processing animage comprising the steps of: -receiving an input image having aplurality of pixels, wherein each of the plurality of pixels have one ormore pixel characteristics; and, -processing the input image to generatean enhanced image by applying a pixel/image relationship to each of theplurality of pixels of the input image, wherein the pixel/imagerelationship is arranged to adjust the one or more pixel characteristicsof each of the plurality of pixels of the input image.

In this embodiment, the method for processing an image is arranged tooperate on a computer system or computing device so as to receive animage, either as a single image in an image file or signal or as animage from a stream of images such as a video or multimedia recordingfrom an image source into an enhanced image. The computer or computersystem may be any type of computer, such as a personal computer,computer server, portable computer, tablet computer, smart phones or acomputer system integrated with other devices such as cameras,televisions, communications/broadcasting equipment. The image source mayalso be a camera module, a communications gateway, an image signalbroadcaster/receiver or a storage device, either remote or local.

As shown in FIG. 1, there is illustrated a schematic diagram of acomputing device which in this embodiment is a computer server 100. Theserver 100 comprises suitable components necessary to receive, store andexecute appropriate computer instructions. The components may include aprocessing unit 102, read-only memory (ROM) 104, random access memory(RAM) 106, and input/output devices such as disk drives 108, inputdevices 110 such as an Ethernet port, a USB port, etc. Display 112 suchas a liquid crystal display, a light emitting display or any othersuitable display and communications links 114. The server 100 includesinstructions that may be included in ROM 104, RAM 106 or disk drives 108and may be executed by the processing unit 102. There may be provided aplurality of communication links 114 which may variously connect to oneor more computing devices such as a server, personal computers,terminals, wireless or handheld computing devices. At least one of aplurality of communications link may also be connected to an externalcomputing network through a telephone line or other type ofcommunications link.

The server 100 may include storage devices such as a disk drive 108which may encompass solid state drives, hard disk drives, optical drivesor magnetic tape drives. The server 100 may use a single disk drive ormultiple disk drives. The server 100 may also have a suitable operatingsystem 116 which resides on the disk drive or in the ROM of the server100.

The server 100 is arranged to have instructions, in the form ofsoftware, hardware or a combination of both which may be executed by theserver to provide an example of method for processing an image. Theserver, when arranged to have these instructions may then operate as asystem for processing images as shown in FIG. 2. Embodiments of theserver 100, may be implemented to process an image file or signal, andmay be implemented as a computer or as a dedicated processing devicewhich can operate in a smart phone or together with image acquiring orbroadcasting equipment such as cameras or image broadcasting systems ortelevision screens.

The computer server 100, which may also be implemented on a standalonecomputer, portable computer, smart device, or as a logical computingcircuit that operates with an electronic or communication device isarranged to receive an input image or input image signal for processing.Once received, the input image or signal is then processed with an imageenhancement engine which may be implemented on a computer or computersystem in hardware, software or a combination of both. The imageenhancement engine is arranged to operate an enhancement process, whichmay include, for example, a image manipulation process, on the inputimage to generate an enhanced image. This enhancement process mayinclude the application of a pixel/image relationship to the inputimage. Embodiments of the pixel/image relationship will be describedfurther below with reference to FIGS. 2 to 5, but in brief, therelationship represents a mapping relationship between thecharacteristics of each individual pixel of the input image against theentire input image itself. Accordingly, by formulating this relationshipinto a model or function that may be applied by a processor to processimage data, adjustments to the characteristics of each individual pixelof the input image may be made so as to affect, and preferably enhance,the entire input image itself.

Preferably, the adjustment of the characteristics of each pixel withinan image may include the adjustment of any characteristics of the pixel,such as lighting intensity or color. In turn, this may affect theexposure levels of one of the individual pixel of the input image. Whenthis adjustment is made to all of the pixels in the image, and thesecombined adjustments are made based on the optimizing of the exposurelevels or other characteristics of the pixel with respect to the entireimage, then in turn, the image is enhanced and becomes an enhancedimage. This example is particularly advantageous as images that may havebeen captured, stored or broadcast with a lower dynamic range can beprocessed and enhanced into one of higher dynamic range, allowing theimage quality to be improved and made suitable for television or viewingscreen with a higher specification that that of the capture, store orbroadcasting equipment.

In some example embodiments of the image enhancement engine, by using anexample of a pixel/image relationship, the enhancement engine maytherefore be able to adjust the pixels of a standard dynamic range (SDR)image or signal so as to enhance the SDR image or signal into a highdynamic range (HDR) image or signal. This adjustment may be performed byincreasing the dynamic range of the image through the enhancementsperformed which would allow details that could not be shown due to theimage's dynamic range limitation to be shown after it has been enhanced.Examples of the pixel/image relationship will be described further belowwith reference to FIGS. 2 to 5.

With reference to FIG. 2, there is shown a system for processing animage 200 comprising:

-   -   an image gateway arranged to receive an input image 202 having a        plurality of pixels, wherein each of the plurality of pixels        have one or more pixel characteristics; and,    -   an enhancement engine 206 arranged to process the input image        202 to generate an enhanced image 208 by applying a pixel/image        relationship to each of the plurality of pixels of the input        image, wherein the pixel/image relationship is arranged to        adjust the one or more pixel characteristics of each of the        plurality of pixels of the input image.

In this embodiment, the system is firstly arranged to receive an inputimage 202 from an image source such as a camera or any image stream 204or storage device. This input image may exist as an image file, an imagedata stream or a signal. Once an input image 202 is received via agateway (not shown) of the system 200 which is any device orcommunication port arranged to communicate with the image source, theinput image 202 is processed by an enhancement engine 206 so as toenhance the image into an enhanced image 208, which may be a stand-aloneimage 208 or part of an enhanced image stream 210.

In the examples described below, the enhancement engine 206 may takemany forms, but in this embodiment, the enhancement engine 206 uses apixel/image relationship arranged to map each pixel's characteristics tothe overall image itself, and whereby adjustment to these pixel'scharacteristics with respect to the pixel/image relationship allows theoverall image to be manipulated or adjusted visually. One suchadjustment to the image may be to the exposure level of the image, whichwhen performed in an optimized manner, may in turn manipulate the imageto achieve a greater dynamic range. This in turn allows under exposeddetails to be displayed or revealed to a user, whilst details lost orwashed out in over exposed portions of the image may also be adjustedsuch that is visible to the user.

The adjustment to the pixel's characteristics with respect to thepixel/image relationship may also be optimized by use of machinelearning tools and/or loss correction methods. In this example, alearning network is used to perform this adjustment and thereby, whenthe learning network is adequately trained, the adjustment wouldtherefore be made to each of the pixels in the image to enhance theimage. Preferably, the learning network may also be arranged to eitherlearn from training referenced data sets or where reference data setsare not available, by use of one or more image quality loss functions.Examples of the learning network are further described below withreference to FIGS. 3 to 5.

With reference to FIG. 3, there is illustrated a block diagram of anexample image enhancement engine. In this embodiment, the imageenhancement engine 206, or enhancement engine is arranged to process aninput image 202 so as to output an enhanced image 208. The enhancedimage 208 would therefore represent an image that would be the inputimage 202 but having its image characteristics improved. Thisimprovement may include, for example, enhancing the exposure level ofthe image and/or expanding the dynamic range of the image. In turn, thismay improve the appearance of the image by showing greater clarity ordetails.

In one embodiment, the image enhancement engine 206 may be used toconvert standard dynamic range (SDR) signal to high dynamic range (HDR)signal by use of machine learning (e.g. deep learning) based methods.However, so as to accommodate for situations where there are noreferences for supervised training, which may occur in various practicaldeployments where training data is difficult to collect, the imageenhancement engine 206 may also be arranged to operate without anyreference data or references for supervise learning.

Preferably, the enhancement engine 206 includes an image manipulationprocessor 304 which applies a pixel/image relationship on an input imageso as to enhance the input image. The pixel/image relationshipcharacterizes a relationship between a pixel's characteristics and theentire image (comprising a totality of all of the pixels within theimage) itself. This relationship, in some examples, represents aninterdependent or correlating function between one pixel of the inputimage and its effect on the rest of the image such that when thecharacteristics of this pixel is adjusted in any way with respect tothis relationship, the image characteristics of the entire image,including its aesthetic appearances may also change.

Preferably, and as the inventors have devised, examples of thepixel/image relationship may be presented mathematically, or visualized,as a “curve” or in an embodiment, as a parameter-adjustable quadraticcurve. In turn, this curve may be used, to convert a SDR signal to a HDRsignal. Preferably, this relationship or curve may also consider thevalue ranges, curve monotonicity, and gradient backpropagation inConvolutional Neural Networks (CNNs) such that it can be adjusted bychanging the values of specific parameters within the curve. Examples ofthis quadratic curve may also progressively approximate higher-ordercurves by iteratively applying itself, thus it can obtain bettercapability of adjusting the dynamic range of an image.

The pixel/image relationship may be arranged to include at least oneadjustable parameter which can be adjusted to determine how therelationship can be applied on to the image to generate a desiredeffect. This parameter may also be adjusted by a learning network suchas by a lightweight CNN which may be designed to learn the pixel-wiseand best-fitting curve parameters when only the SDR signal (images orvideos) are used for training. In turn, when the learning network isable to adjust the application of the relationship on an image byestimating the necessary parameters, the characteristics of the imagewill change either to improve its appearance via adjustments to variouscharacteristics such as exposure levels, dynamic range, contrast,saturations, etc. Once the learning network is trained to perform suchoptimizing adjustments, it will therefore be able to adjust theparameters to apply the relationship onto an input image withimprovements to the appearance of the image.

In situations where there is an inadequate amount of training data toperform any significant training of the learning network, one or moreimage quality loss functions, which may be non-reference based imagequality loss functions or image quality assessment loss functions, mayalso be used to drive the best-fitting curve parameters learning withoutreferences. In this regard, experimental results performed by theinventors on the enhancement engine show that the engine 206, and whenit is used in a system for processing an image 200, may effectivelyimplement the conversion from SDR signal to HDR signal on synthetic andreal SDR signal in qualitative and quantitative metrics. Furthermore,the enhancement engine 206 is also fast for practical applications (e.g.500 FPS) and can convert SDR signal to HDR signal for displaying on HDRTVs in real time, which can improve the quality of displays of oldermedia on modern screens.

As shown in FIG. 3, the enhancement engine 206 includes an imagecharacteristics estimation network 300 arranged to provide an estimateof image characteristics parameters 302 that may be used to best processthe input image so as to enhance the image in accordance with apixel/image relationship. These parameters may also be referred to (asearlier mentioned) as best fitting curve parameters or parameters. Theparameters 302 estimated will be based on each an input image 202 andevery input image is likely to have its own set of estimated parameters302.

Once these image characteristics parameters 302 are estimated, the imagecharacteristics parameters 302 are then inputted into an imagemanipulation process 304 which would in turn start and control theapplication of a pixel/image relationship to each of the pixels of theinput image to generate an enhanced image. This application process ofthe pixel/image relationship may also be performed iteratively (306) soas to obtain a preferred or optimized enhanced image 208 and uses theestimated parameters 302 to determine how the pixel/image relationshipis applied to the individual pixel concerned.

As shown in this example, an input image 202, which may be in the formof an SDR signal (or any other image or a frame from a stream of imagessuch as a video), is first forwarded to the image characteristicsestimation network 300 to estimate a set of best-fitting parameters 302for applying the image/pixel relationship. These parameters 302 may alsobe referred to as the estimated image characteristics parameter or“curve parameters” where the image/pixel relationship is referred to asa curve, would then determine how the image/pixel relationship isapplied onto the pixel concerned so as to attempt to provide the bestenhancement of the input image.

The image manipulation process 304 proceeds to use the estimated imagecharacteristics parameters 302 or curve parameters and apply thepixel/image relationship in accordance with these parameters to theinput image or signal. In one example, the input pixels of the RGBchannels are mapped to the enhanced level by iteratively (306) applyingthe image/pixel relationship that would operate as a “mapping curve”.Examples of this embodiment relationship or the “mapping curve”, imagecharacteristics parameter estimation network 300, and non-reference lossare further described below.

As described above, the pixel/image relationship may be a function ormathematical relationship which would map the relationship between thecharacteristics of each individual pixel of the image to the entireimage as a whole. In an abstract comparison and to appreciate an exampleof this pixel/image relationship, one can compare the pixel/imagerelationship with the “curves” that may be found in certain “curveadjustment tool” as found in some advanced computerized photo editingsystems. In these curve adjustment tools, a user may be able tomanipulate a curve which is plotted out against an image histogram andin turn manipulate the appearance of the image by tuning variouscharacteristics, such as contrast, exposure, lighting effects,saturation of colors etc. There are many effects of the curve adjustmenttool, but in general, the discrete adjustment of the curve by a user mayalter the exposure of some portions of the image and in turn, whenmanipulated by a user, using the image as a feedback, the user can“tune” the exposure, lighting, contrast and visible level of detail ofthe image as the user pleases so as to present the most aestheticaloptimal image.

In this regard, the pixel/image relationship is comparable in anabstract sense to this “curve” in the curve adjustment tool, as itsmanipulation, preferably, by automation and not human intervention,would also allow the exposure or other image characteristics, to beadjusted and optimized. In this regard, the pixel/image relationship maybe able to map a low-light image to its enhanced version automaticallyand that the pixel/image relationship is solely dependent on the inputimage. Nonetheless, it should be appreciated by persons skilled in theart that the pixel/image relationship described herein is fundamentallydifferent in all aspects when compared with examples of the curveadjustment tools found in photo editing systems as the pixel/imagerelationship is devised by use of different variables with entirelydifferent mathematically relationships and representations.

Preferably, in defining the pixel/image relationship, there are threeobjectives for of the pixel/image relationship. These are as follows:

-   -   1) each pixel value of the converted HDR signal should be in the        normalized range, such as [0,1], which avoids the information        loss induced by overflow truncation;    -   2) this relationship should be monotonous to preserve the        differences of neighboring pixels;    -   3) the form of this relationship should be as simple as possible        and meet the requirement of gradient backpropagation. For the        brief description, an SDR image is taken as an example as        follows.

To achieve these three objectives, an example of the image/pixelrelationship may be represented mathematically as a quadratic curvewhich goes across zero and one points may be expressed as:F(I(x);α)=I(x)+αI(x)(1−I(x))   (1)where

-   -   x denotes the pixel coordinates,    -   F(I(x);α) is the enhanced result (such as a corresponding HDR        image) of an input SDR image,    -   I(x), α∈[−1,1] is the trainable curve parameter, which adjusts        the magnitude of curve; and    -   Each pixel is normalized to [0,1] and all operations are        pixel-wise.

With reference to FIG. 3A, there is shown a number of charts (a) to (e)showing the curves with different adjustment parameters α∈[−1,1] and thenumber of iterations n∈{1,2,3,4}. As shown in FIG. 3A, the form ofcurves is illustrated with different adjustment parameters a in FIG.3A(a). The curve enables the capability of increasing and decreasing thedynamic ranges of input image, which is conducive to not only enhancingthe dark or low light regions (e.g. SDR regions) but also avoiding theover-exposure artefacts.

In several challenging examples, this curve should have a more powerfuladjustment capability. To the end, the higher-order curves areapproximated by iteratively using the quadratic curve as follows:F _(n)(x)=F _(n−1)(x)+α_(n) F _(n−1)(x)(1−F _(n−1)(x))   (2)

where

-   -   n is the number of iterations.    -   Eq. (2) can be reduced to Eq. (1) when n is set to 1.

In FIGS. 3(b)-(e), the higher-order curves with different a and n arepresented. For example, the curve with n=4 and α_(1,2,3,4)=1 as thegreen (g) curve shown in FIG. 3A(e) has more powerful adjustmentcapability than the basic LE-curve with α₁=1 as the green (g) curveshown in FIG. 3(a). In one experiment example, the inventors set n to 8,which can deal with most of the cases.

Although such a higher-order curve can adjust the image in the largedynamic range, it is still a global adjustment since the a is used forall pixels. The global mapping tends to over/under enhance localregions. To solve this problem, the a is formulated as the pixel-wiseparameter. It means each pixel of input image has a corresponding curvewith the best-fitting a to adjust its dynamic range. Hence, the final ais a parameter map with the same size as the input image, and Eq. (2)can be further expressed as:F _(n)(x)=F _(n−1)(x)+α_(n)(x)F _(n−1)(x)(1−F _(n−1)(x))   (3)

In this regard, the assumption is that the pixels in the local regionhave the same intensity (also the same adjustment curves); thus, theneighbouring pixels in the enhanced result still preserve the monotonousrelations. In this manner, the higher-order curves also comply with theabove-mentioned objectives. With the best-fitting maps, the enhancedresult can be directly obtained by curve mapping.

With reference to FIG. 3B, there is illustrated an example of an imagecharacteristics parameter estimation network arranged to estimate a setof best-fitting curve parameter maps of three channels (R, B, G) of aninput image 3B(a). As shown in FIG. 3B, the best-fitting parameter mapsin different channels 3B(b), 3B(c), 3B(d) have similar adjustmenttendency but different values. Moreover, the curve parameter mapaccurately indicates the brightness of different regions (e.g, the twoglitters on the wall). With the best-fitting maps, the enhanced resultcan be directly obtained (e.g., FIG. 3B(e)) which reveals the content inthe dark regions and well preserves the regions of light source.

With reference to FIG. 4, there is illustrated an example embodiment ofimage characteristics parameter estimation network 300 of FIG. 3. Asdescribed earlier, in order for the image manipulation processor 304 toapply the pixel/image relationship to each pixel of the input image, itis necessary to identify the parameters 302 which will be used to applythe pixel/image relationship to each pixel. Accordingly, in thisexample, to build the relations between the input image 202 and itsoptimized application of the pixel/image relationship, or as visualizedas building the best-fitting curve parameter maps, an imagecharacteristics parameter estimation network 300 (parameter estimationnetwork) is used to estimate these parameters.

As shown in FIG. 4, the input to the parameter estimation network 300 isthe input image 202, which may be an SDR image or signal while theoutputs are a set of curve parameter maps for corresponding higher-ordercurves. With the curve parameter maps, the enhanced result (HDR image)can be obtained by using Eq. (3).

In one example, the parameter estimation network 300 is a learningnetwork that can be trained. Preferably, instead of employing fullyconnected layers which require the fixed input sizes, a convolutionneural network (CNN) for the estimation of curve parameter maps may beemployed. The network 300 may not use the down-sampling and batchnormalization layers which break the relations of neighboring pixels. Anexample of the detailed architecture of this curve parameter estimationnetwork 300 is shown in FIG. 4. Specifically, in this example, onlyseven convolutional layers with symmetrical concatenation are used. Eachlayer consists of 32 convolutional kernels of size 3*3 and stride 1followed by ReLU function. The last convolutional layer with Tanhfunction outputs 24 parameter maps according to the numbers of iteration(e.g., n=8 in one experiment), where each iteration requires 3 curveparameter maps for three channels. In turn, this network 300 may betrained with a reference dataset 400. However, a reference data set 400may not always be available in which case, one or more image qualityloss functions or image quality assessment loss functions 402 may beused to provide training to the network.

As illustrated in FIG. 4, there is the learning network has one or moreimage quality assessment loss functions 402 arranged to drive thelearning of curve parameter estimation network 300. These loss functions402 predict a loss or error for the estimate parameters 302 by assessingthe image quality for specific quality losses after it has been adjustedby the image manipulation processor 304. In turn, the functions are ableto provide a feedback mechanism to train the curve parameter estimationnetwork 300. These loss functions 402 may include spatial consistencyloss (404), exposure control loss (406), color constancy loss (408), andillumination smoothness loss (410). Each of these are described below.

Spatial Consistency Loss (404)

The enhanced result is expected to inherit the spatial consistency fromthe input image. In other words, the bright (dark) regions should keeprelatively bright (dark) in the enhanced result. To implement thespacial consistency loss L_(spa), the difference of neighboring regionsin input and enhanced images is computed as:

$\begin{matrix}{L_{spa} = {\frac{1}{K}{\sum\limits_{i = 1}^{K}\;{\sum\limits_{j \in {\Omega{(i)}}}\left( {{{Y_{i} - Y_{j}}} - {{I_{i} - I_{j}}}} \right)^{2}}}}} & (4)\end{matrix}$

where K is the number of cantered region i, Ω(i) is the four neighboringregions cantered at the region i. Y and I represent the averageintensity value of the local region in the enhanced result and inputimage, respectively. The size of the local region is set to 4*4.

Exposure Control Loss (406)

To restrain the under/over-exposed regions, an exposure control loss isdevised to control the exposure level. Firstly, the average intensityvalue in each nonoverlapping local region of the enhanced image iscomputed. Then the average intensity value subtracts a predefinedwell-exposedness level E. The exposure control loss L_(exp) measures howclose the average intensity value of the local region is close to thewell-exposedness level, and can be expressed as:

$\begin{matrix}{L_{\exp} = {\frac{1}{M}{\sum\limits_{k = 1}^{M}\;{{Y_{k} - E}}}}} & (5)\end{matrix}$

where M represents the number of the nonoverlapping local region, Y isthe average intensity value of a local region in the enhanced image. Thesize of the local region and well-exposedness level are set to 16*16 and0.6, respectively.

Color Constancy Loss (408)

Following Gray-World color constancy hypothesis that color in eachsensor channel averages to gray over the entire image, a color constancyloss is invented to correct the potential color deviations in theenhanced result and also build the relations among three channelsseparate adjustment. The color constancy loss L_(col) is to ensure theaverage intensity values of RGB channels close, which can be expressedas:

$\begin{matrix}{{L_{col} = {\sum\limits_{\forall{{({p,q})} \in {\mathcal{s}}}}\left( {J^{p} - J^{q}} \right)^{2}}},{ɛ = \left\{ {R,G,B} \right\}}} & (6)\end{matrix}$

where J^(p) denotes the average intensity value of p channel of theenhanced result, (p,q) represents a pair of color channel.

Illumination Smoothness Loss (410) To preserve the monotonicityrelations between the neighboring pixels, an illumination smoothnessloss to each curve parameter map α is invented. The illuminationsmoothness loss L_(tvα) is defined as:

$\begin{matrix}{{L_{{tv}\;\alpha} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}\;{\sum\limits_{c \in {\mathcal{s}}}\left( {{\nabla_{x}\alpha_{n}^{c}} + {\nabla_{y}\alpha_{n}^{c}}} \right)^{2}}}}},{ɛ = \left\{ {R,G,B} \right\}}} & (7)\end{matrix}$

where N is the number of iterations, a is the curve parameter map, ∇_(x)and ∇_(y) represent the horizontal and vertical gradient operations.

The total loss is therefore a linear combination of the aforementionedlosses:L _(total) =L _(spa) +L _(exp) +W _(col) L _(col) +W _(tvα)L_(tvα)  (8)

Each of these image quality loss functions 402 may be used individuallyor in combination to provide a loss value that can be used to train theimage characteristics estimation network 300, or any other machinelearning tool or learning network that is used in its place.

In one example implementation, in order to perform the training processof the network 300, any low image quality image, such as an SDR image orvideo datasets can be used for training. Preferably, the training imagesare firstly resized to an example 512*512 although other alternativesizes also can be used. The system for processing an image 200 isimplemented with computing device having a GPU. A batch-mode learningmethod with a batch size of 8 is applied. The filter weights of eachlayer are initialized with standard zero mean and 0.02 standarddeviation Gaussian function. Bias is initialized as a constant.

An ADAM optimizer with default parameters and fixed learning rate 0.0001may also be used for this network optimization. The weights W_(col) andW_(tvα) are set to 0.5, and 20, respectively. With an GPU such as the(NVIDIA GTX 2080Ti GPU) as an example, the proposed framework canprocess an image with a size of 640*480 within 0.002 s (500 FPS).

The image quality loss functions 402 may be advantageous as they areable to consider the quality of the image from a number of differentlight or image related effects. In this regard, each specific function(404), (406), (408) and (410) has been tested by the inventors in theirtrials and experiments so as to identify and measure the manner in whicheach of these loss functions contribute to the enhancement of an image.

With reference to FIG. 5, the results of combining various lossfunctions are shown. In these experiments, an image as shown in (a) isinputted into an example embodiment of a system for processing an imageas described in FIGS. 2 to 4, with an enhanced image result of (b).

In this regard, the inventors performed an ablation experiment byremoving each of the loss functions 402 whilst combining the remainingloss functions. As indicated in FIG. 5(a) to (e), when each lossfunction is selectively removed, the enhancement to the image suffers aspecific effect which would render the enhancement to be poorlyperformed.

When compared with (b), (c) is the result without spatial consistencyloss L_(spa) (404) which shows it has relatively low contrast, such asthe region of cloud, as without this loss function, the image losses thedifference between neighboring regions existed in the input.

When removing the exposure control loss L_(exp) (406), as shown in (d),the brightness of input has less change, which indicates the importanceof exposure control loss for lowlight image adjustment.

The result in (e) introduces color casts when the color constancy lossL_(col) (408) is discarded. Such a framework ignores the relationsbetween three channels when separately using curve mapping.

In (f), removing the illumination smoothness loss L_(tva) (410) greatlydamages the correlations between neighboring regions and thus introducesobvious artifacts.

The results in FIG. 5 would indicate that it is preferable to combineall of the loss functions (404), (406), (408) and (410) as thecontribution of each function is clearly demonstrated in thisexperiment.

However, depending on the exact implementation, training set, learningnetwork conditions and the input image itself, it is expected that notall of the loss functions (404), (406), (408) and (410) need to be used.In certain circumstances, one or a combination of any one or more ofthese functions, may also provide an acceptable level of performance.

Embodiments of the present invention may be advantageous for at leastthe various advantages.

(1) In a first aspect, the system provides for an example of azero-reference learning framework for converting SDR signal to HDRsignal. Additionally, it is independent on the paired and unpairedtraining data, and thus avoids the risk of overfitting on specific data.As a result, this framework generalizes to various SDR signal.

(2) The system provides for an example of an image/pixel relationshipwhich may be represented by a quadratic curve. This curve may be able toapproximate higher-order curves by iteratively applying itself. Insteadof image reconstruction used in deep learning-based SDR signal to HDRsignal approaches which may damage the intrinsic attributes of an inputSDR signal, the devised image-specific curve can self-adaptively adjustSDR signal to its corresponding HDR signal by pixel-level mapping.

(3) The system also includes a learning network which operates as alightweight curve parameter estimation network. The network is easy totrained (0.5 hours in example experiments) and fast for inference, whichis suitable for practical applications.

(4) The learning network of the system is also able to avoid anyreliance on reference data, a new task-specific non-reference lossfunction including spatial consistency loss, exposure control loss,color constancy loss, and illumination smoothness loss is devised.

(5) Accordingly, a zero-reference learning framework can be extended toother image processing tasks.

Although not required, the embodiments described with reference to theFigures can be implemented as an application programming interface (API)or as a series of libraries for use by a developer or can be includedwithin another software application, such as a terminal or personalcomputer operating system or a portable computing device operatingsystem. Generally, as program modules include routines, programs,objects, components and data files assisting in the performance ofparticular functions, the skilled person will understand that thefunctionality of the software application may be distributed across anumber of routines, objects or components to achieve the samefunctionality desired herein.

It will also be appreciated that where the methods and systems of thepresent invention are either wholly implemented by computing system orpartly implemented by computing systems then any appropriate computingsystem architecture may be utilised. This will include standalonecomputers, network computers and dedicated hardware devices. Where theterms “computing system” and “computing device” are used, these termsare intended to cover any appropriate arrangement of computer hardwarecapable of implementing the function described.

It will be appreciated by persons skilled in the art that numerousvariations and/or modifications may be made to the invention as shown inthe specific embodiments without departing from the spirit or scope ofthe invention as broadly described. The present embodiments are,therefore, to be considered in all respects as illustrative and notrestrictive.

Any reference to prior art contained herein is not to be taken as anadmission that the information is common general knowledge, unlessotherwise indicated.

The invention claimed is:
 1. A method for processing an image comprisingthe steps of: receiving an input image having a plurality of pixels,wherein each of the plurality of pixels have one or more pixelcharacteristics; and, processing the input image to generate an enhancedimage by applying a pixel/image relationship to each of the plurality ofpixels of the input image, wherein the pixel/image relationship isarranged to adjust the one or more pixel characteristics of each of theplurality of pixels of the input image; wherein the pixel/imagerelationship is applied iteratively to each of the plurality of pixelsin the processing step to generate the enhanced image.
 2. A method forprocessing an image in accordance with claim 1, wherein the pixel/imagerelationship is associated with a relationship between the one or moreof characteristics of each of the plurality of pixels of the input imageand a visual presentation of the input image.
 3. A method for processingan image in accordance with claim 2, wherein the one or morecharacteristics of each of the plurality of pixels are associated withthe exposure levels of each of the plurality of pixels.
 4. A method forprocessing an image in accordance with claim 3, wherein the pixel/imagerelationship includes a mapping of the one or more characteristics ofeach of the plurality of pixels relative to the one or morecharacteristics of each of the other pixels of the input image.
 5. Amethod for processing an image in accordance with claim 4, wherein theexposure levels include tones, contrasts or color shifts of the pixel.6. A method for processing an image in accordance with claim 1, whereinby applying the pixel/image relationship to each of the plurality of thepixels of the input image, a dynamic range of the input image isadjusted.
 7. A method for processing an image in accordance with claim1, wherein the pixel/image relationship includes at least one adjustableparameter arranged to adjust a magnitude of the application of thepixel/image relationship to each of the plurality of pixels.
 8. A methodfor processing an image in accordance with claim 7, wherein themagnitude of the application of the pixel/image relationship applies tothe one or more characteristics of each of the plurality of pixels.
 9. Amethod for processing an image in accordance with claim 7, wherein theat least one adjustment parameter is adjustable by a learning network.10. A method for processing an image in accordance with claim 9, whereinthe learning network is a convolution neural network (CNN).
 11. A methodfor processing an image in accordance with claim 9, wherein the learningnetwork is trained with a reference data set.
 12. A method forprocessing an image in accordance with claim 9, wherein the learningnetwork is trained with one or more image quality loss processes.
 13. Amethod for processing an image in accordance with claim 12, wherein theone or more image quality loss processes include spatial consistencyloss, exposure control loss, color constancy loss, illuminationsmoothness loss or any combination thereof.
 14. A method for processingan image in accordance with claim 12, wherein the one or more imagequality loss processes are used when the reference data set is notavailable.
 15. A method for processing an image in accordance with claim1, wherein the image/pixel relationship is represented byF(I(x);α)=I(x)+αI(x)(1−I(x)) where x denotes the pixel coordinates,F(I(x);a) is the enhanced image, I(x), αϵ[−1,1] is the adjustableparameter and wherein each pixel is normalized to [0,1].
 16. A methodfor processing an image in accordance with claim 13, wherein the spatialconsistency loss is represented by$L_{spa} = {\frac{1}{K}{\sum\limits_{i = 1}^{K}\;{\sum\limits_{j \in {\Omega{(i)}}}\left( {{{Y_{i} - Y_{j}}} - {{I_{i} - I_{j}}}} \right)^{2}}}}$where K is the number of cantered region i, Ω(i) is the four neighboringregions cantered at the region i. Y and I represent an average intensityvalue of a local region in the enhanced result and input image,respectively.
 17. A method for processing an image in accordance withclaim 13, wherein the exposure loss control is represented by:$L_{\exp} = {\frac{1}{M}{\sum\limits_{k = 1}^{M}\;{{Y_{k} - E}}}}$where M represents the number of nonoverlapping local regions, Y is theaverage intensity value of a local region in the enhanced image.
 18. Amethod for processing an image in accordance with claim 13, wherein thecolor Constancy Loss is represented by${L_{col} = {\sum\limits_{\forall{{({p,q})} \in {\mathcal{s}}}}\left( {J^{p} - J^{q}} \right)^{2}}},{ɛ = \left\{ {R,G,B} \right\}}$where J^(p) denotes the average intensity value of p channel of theenhanced result, (p,q) represents a pair of color channel.
 19. A methodfor processing an image in accordance with claim 13, wherein theillumination smoothness loss is represented by${L_{{tv}\;\alpha} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}\;{\sum\limits_{c \in {\mathcal{s}}}\left( {{\nabla_{x}\alpha_{n}^{c}} + {\nabla_{y}\alpha_{n}^{c}}} \right)^{2}}}}},{ɛ = \left\{ {R,G,B} \right\}}$where N is the number of iterations, α is the curve parameter map,∇_(x)and ∇_(y)represent the horizontal and vertical gradient operations.20. A method for processing an image in accordance with claim 1, whereinthe input image is a Standard Dynamic Range (SDR) signal and theenhanced image is a High Dynamic Range (HDR) signal.
 21. A system forprocessing an image comprising: an image gateway arranged to receive aninput image having a plurality of pixels, wherein each of the pluralityof pixels have one or more pixel characteristics; and, an enhancementengine arranged to process the input image to generate an enhanced imageby applying a pixel/image relationship to each of the plurality ofpixels of the input image, wherein the pixel/image relationship isarranged to adjust the one or more pixel characteristics of each of theplurality of pixels of the input image; wherein the pixel/imagerelationship is applied iteratively to each of the plurality of pixelsto generate the enhanced image.
 22. A system for processing an image inaccordance with claim 21, wherein the pixel/image relationship isassociated with a relationship between the one or more characteristicsof each of the plurality of pixels of the input image and a visualpresentation of the input image.
 23. A system for processing an image inaccordance with claim 22, wherein the one or more characteristics ofeach of the plurality of pixels are associated with the exposure levelsof each of the plurality of pixels.
 24. A method for processing an imagein accordance with claim 23, wherein the pixel/image relationshipincludes a mapping of the one or more characteristics of each of theplurality of pixels relative to the one or more characteristics of eachof the other pixels of the input image.
 25. A system for processing animage in accordance with claim 24, wherein the exposure levels includetones, contrasts or color shifts of the pixel.
 26. A system forprocessing an image in accordance with claim 21, wherein by applying thepixel/image relationship to each of the plurality of the pixels of theinput image, a dynamic range of the input image is adjusted.
 27. Asystem for processing an image in accordance with claim 21, wherein thepixel/image relationship includes at least one adjustable parameterarranged to adjust a magnitude of the application of the pixel/imagerelationship to each of the plurality of pixels.
 28. A system forprocessing an image in accordance with claim 27, wherein the magnitudeof the application of the pixel/image relationship applies to the one ormore characteristics of each of the plurality of pixels.
 29. A systemfor processing an image in accordance with claim 27, wherein the atleast one adjustment parameter is adjustable by a learning network. 30.A system for processing an image in accordance with claim 29, whereinthe learning network is a convolution neural network (CNN).
 31. A systemfor processing an image in accordance with claim 29, wherein thelearning network is trained with a reference data set.
 32. A system forprocessing an image in accordance with claim 29, wherein the learningnetwork is trained with one or more image quality loss processes.
 33. Asystem for processing an image in accordance with claim 32, wherein theone or more image quality loss processes include spatial consistencyloss, exposure control loss, color constancy loss, illuminationsmoothness loss or any combination thereof.
 34. A system for processingan image in accordance with claim 32, wherein the one or more imagequality loss processes are used when the reference data set is notavailable.
 35. A method for processing an image comprising the steps of:receiving an input image having a plurality of pixels, wherein each ofthe plurality of pixels have one or more pixel characteristics; and,processing the input image to generate an enhanced image by applying apixel/image relationship to each of the plurality of pixels of the inputimage, wherein the pixel/image relationship is arranged to adjust theone or more pixel characteristics of each of the plurality of pixels ofthe input image; wherein a learning network is used to adjust the one ormore pixel characteristics of each of the plurality of pixels of theinput image, and the learning network is trained with one or more imagequality loss processes including spatial consistency loss, exposurecontrol loss, color constancy loss, illumination smoothness loss or anycombination thereof.
 36. A method for processing an image comprising thesteps of: receiving an input image having a plurality of pixels, whereineach of the plurality of pixels have one or more pixel characteristics;and, processing the input image to generate an enhanced image byapplying a pixel/image relationship to each of the plurality of pixelsof the input image, wherein the pixel/image relationship is arranged toadjust the one or more pixel characteristics of each of the plurality ofpixels of the input image; wherein a learning network is used to adjustthe one or more pixel characteristics of each of the plurality of pixelsof the input image; the learning network trained with one or more imagequality loss processes when a reference data set is not available fortraining the learning network.
 37. A method for processing an imagecomprising the steps of: receiving an input image having a plurality ofpixels, wherein each of the plurality of pixels have one or more pixelcharacteristics; and, processing the input image to generate an enhancedimage by applying a pixel/image relationship to each of the plurality ofpixels of the input image, wherein the pixel/image relationship isarranged to adjust the one or more pixel characteristics of each of theplurality of pixels of the input image; wherein the pixel/imagerelationship is represented byF(I);α)=I(x)+αI(x)(1−I(x)) where x denotes the pixel coordinates,F(I(x);α) is the enhanced image, I(x), αϵ[−1,1] is the adjustableparameter and wherein each pixel is normalized to [0,1].